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Abstract: Selecting elite students out of a huge collective is a difficult task. The main problem is to provide 
automated processes to reduce human work. ECMS (Educational Contest Management System) is an online tool 
approach to help - fully or partly automated - with the task of selecting such elite students out of a mass of 
candidates. International tests like the PISA study revealed the need for such new methods in extracting better 
students out of collectives and to offer new online based tools for knowledge gaining processes. 
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1. Introduction 

The problem of selecting elite students out of a 
collective is the need of massive human 
resources to organize and operate such a 
process. Automating these processes is an 
essential task to reduce time and human 
efforts. The need for such tool applications 
increased after several statistical studies and 
publications like the PISA Study (Stanat 2000; 
OECD 2004a). Since the national results of the 
PISA Study (OECD 2004b) have been a 
shocking event, a wide spread discussion 
broke out about how to solve such educational 
problems and what tools would be necessary 
to teach and test scholars and students more 
efficient. 

Unlike many existing test and assessment 
tools (TA) or LMS (Learning Management 
Systems) - like the Blackboard Learning 
System (Blackboard 2004), ILIAS (ILIAS 
2004), Moodle (Moodle 2004), Claroline 
(Claroline 2004), .LRN (Bartle 2002; MIT Sloan 
2002) or the Whiteboard Courseware System 
(Whiteboard 2004) - the introduced ECMS 
(Educational Contest Management System) 
provides not only an e-Learning platform. It 
also gives opportunities to offer simple multi¬ 
choice tests and homework 
problems(Schneider 2004a). An essential part 
of ECMS is the semester-based structure of 
exams to be sat by the students. This allows 
for the scenario that not all students will pass 


each semester (exam) and join the next 
semester (exam). ECMS is now developed as 
version 2.0 and is integrated into the 
Universitas Virtualis (Virtual University 
Environment) project (Schneider 2004b). A 
previous version of ECMS 1.0 (’’The Reverse- 
Engineering-Academy (REA)”) (REA 2002; 
Schneider 2004b) - with focus on software- 
protection (SP) and Reverse Code 
Engineering (RCE) - shows the success of the 
ECMS sieving process in the time range of a 
long time survey of 2 1/2 years using very hard 
problems increasing in difficulty (see section 
(III)). The ECMS has it’s own connection to the 
Universitas Virtualis Bibliotheca Server 
(Schneider 2004c) which uses GDL 4.0 
(Ganesha Digital Library Project) (Fahmi 2004) 
as its core platform and provides additional 
access to learning material and ECMS related 
publications. 

2. System description 

ECMS is based on MySGL 4.0 and PHP 4.3.0. 
It is not dependent on third-party packages 
and can be installed easily on local machines 
or any servers using the Apache Webserver 
System (Behlendorf 2004). The ECMS can be 
accessed via standard web-browsers (See 
figure (2)). Figure (1) shows the UML layout of 
the database scheme. 
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Figure 1: UML Diagram of the ECMS database scheme 


As a basic principle ECMS is capable of 
offering an unlimited mixture of multiple choice 
tests and homework. It is therefore open for 
any type of exam content. ECMS 2.0 currently 
offers a free and anonymous registration 
process for students, which is capable of 
extension and cross-link to third-party 
authorisation systems. Students have access 
to multiple provided course material which can 
be attached as original file (e.g. .doc or .pdf) or 
as a package file (e.g. .zip or .tar.gz) by the 
exam moderator. To prevent students from 
being non-active after the registration process, 
ECMS automatically bans all students not 
finishing the first exam after 30 days and 
deactivates the account to prevent re¬ 


registration with the same userdata. The 
current grading and ranking table can be 
viewed by every registered student. The ECMS 
can only be used by registered students using 
an self-set password. Extending ECMS to 
accept additional user registration data or 
using a different password system is made 
easy and there is an option to adapt the 
system for individual needs. 

To give granted access to the administration 
and moderation area, ECMS uses a simple but 
effective security concept using .htaccess and 
.htpasswd files. Adapting the administration 
authentification process to database driven 
variants can be done quickly and easily. 



Figure 2: The ECMS Web-Frontend offering a multiple choice test. 


2.1 Multiple choice exam layout 

The core of each Multiple Choice Exam is the 
XML based template which definition can be 
seen in figure (3). The template defines the 


name and a description for each given exam to 
add additional information .about the topic or 
similar. <MINCORRECT> defines the minimum 
number of correct answers to pass the exam. 
Failing the exam will result in an overview page 
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showing which answers had been wrong but 
without any information about what was wrong. 
ECMS protects itself against cheating via 
history- back-button brute force attacks. 
<TRIALS> defines the maximum tries which 
are allowed for the student. Exceeding the 
maximum number will lead into automated 
failure and results in automated banning from 
ECMS. After the general exam information 
each multiple-choice question is defined by 
<TESTITEM and a unique identifying id. Using 
these id’s ECMS can build dynamic forms for 
the web front-end. Each test item consists of 
several node informations. <QUESTION> 
contains the question the student will be 

<ItbI> 


asked. <ITEM defines over unique id’s the 
possible pre-set answers of the student. To 
identify the correct answer the <CORRECT> 
tag connects to the corresponding <ITEM id. 
The XML structure gives the possibility to add 
unlimited questions to each exam. The XML is 
stored in the database and will be loaded 
dynamically with each form called by the 
student. <ATTACHEDFILE defines using id an 
unlimited number of attached files of non- 
restricted formats (.pdf, .zip, .doc,) which can 
for example contain files with courseware or 
lectures. To offer additional information the 
<ATTACHEDIMAGE tag contains a link to 
attached images related to the exam. 


<NAME>File Knowledge Exam</NAME> 
<DESCRIPTION> 

This exam contains questions about... 
</DESCRIPTION> 

<MINCORRECT>8</MINCORRECT> 

<TRIALS>6</TRIALS> 


<TESTITEM id="1"> 

<Q> 

<QUESTION> 

How many sections can... 
</QUESTION> 

<ATTACHEDFILE id="1"> 
coursewarel .zip 
</ATTACHEDFILE> 
<ATTACHEDIMAGE id="1"> 
question1.gif 
</ATTACHEDtMAGE> 
<CORRECT>2</CORRECT> 

<ITEM id-'l">4.294.967.295</ITEM> 
<ITEM id="2">65.535</ITEM> 

</Q> 

</TESTITEM> 

</TEST > 

Figure 3: XML template for multiple choice exams. 


2.2 Homework system 

The homework feature consists of giving the 
student the problem and needed material 
which may be needed to solve the problem. 
The solution will be uploaded by the student 
and can be accessed by the administrators 
and moderators in the administration area of 
ECMS. Homework will be checked manually by 
any moderator registered. Since each 
homework is correlated to one exam 
constructor (moderator) an automated mail 
information will be send to the moderator about 
the new incoming homework. If the homework 
fails, the moderator marks the homework as 
failed and uses the ECMS mail functionality to 
tell the student why the solution had been 


denied. To accept the solution the moderator 
klicks the hyperlink ’’upgrade” and the student 
will be automatically set one exam higher 
including an automated mail about the 
accepted status of the solution. Denying a 
homework will activate the same process with 
difference that the moderator can enter the 
reason of the negative answer. Since students 
can not be banned automated within the 
homework process, the moderator has the 
option to ban and unban the student from 
ECMS directly. The number of failures is 
stored in the database and is shown to the 
moderator. 
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2.3 Gathering of elite students via 
statistical functions 

ECMS 2.0 offers an extendable possibility for 
evaluating the running sieving progress. Since 
all data is collected in the main ECMS 
database, statistics can be easily extracted 
from it. Possible evaluation modules can 
contain histograms, scatter-plots or other 
statistical related methods. From such 
statistical methods the contest manager can 
gain an overview on how the progress of the 
contest and the exams is and may react on the 
problems occurring with the exam contents. 

3. Evaluation and results of ECMS 

1.0 

ECMS 1.0 was first introduced in 2002. It’s 
exams were focused on software-protection, 
cryptography and Reverse Code Engineering 
(RCE) topics. ECMS 1.0 included 19 main 
exams which had been all declared as 
homeworks. Additionally there had been 10 
correlated homeworks to improve the student 


knowledge. Another 39 exams with topics 
mathematics and crptography had been added 
as fun part to give students more tasks to 
solve. Solving these 39+10 exams had no 
influence on the sieving process. ECMS 1.0 
gave the possibility to register anonymous via 
an existing e-mail account and an username. 
After 2 years of running the system counted 
6361 registered students in the ECMS system. 
The automated sieving process lead via auto¬ 
ban functionality in banning 4998 students who 
had not been able to solve the first main exam 
within 30 days. Therefore 78.57% of the 
students failed the first main exam. The rest of 
1352 (21.25%) students were capable of 
solving the first main exam within the 30 days 
limit. From the remaining 1352 students only 9 
students had been able to solve all 19 main 
exams. This is 0.14% of the registered 6361 
students and 0.67% of the 1352 active 
students. The average solving time of all 
exams was a 3/4 year. The set of increasing 
difficulties was well chosen for the contest 
focus (See figure (4) and table (1)). 


ECMS version 1 0 - Statistical Distnbuoon of Students vs Exam -Level 



Figure 4: Statistics of the sieving process during 19 exams. 


Tablel shows the statistics of the sieving 
process during 19 exams including percentage 
of total registered students and of students 
passing exam number 1. 


Table 1: Sieving process statistics 


Exam- 

Level 

No. of 
Students 

% of total 

% of after 
exam 1 

19 

9 

0.14 

0.67 

18 

21 

0.33 

1.55 

17 

32 

0.50 

2.37 

16 

44 

0.69 

3.25 

15 

45 

0.71 

3.33 

14 

60 

0.94 

4.44 

13 

83 

1.30 

6.14 

12 

93 

1.46 

6.88 

11 

96 

1.51 

7.10 

10 

113 

1.78 

8.36 

9 

150 

2.36 

11.09 

8 

155 

2.44 

11.46 

7 

162 

2.55 

11.98 


Exam- 

Level 

No. of 
Students 

% of total 

% of after 
exam 1 

6 

174 

2.74 

12.87 

5 

193 

3.03 

14.28 

4 

282 

4.43 

20.86 

3 

355 

5.58 

26.26 

2 

488 

7.67 

36.10 

1 

1352 

21.25 

100.00 


4. Possible Scenarios 

There are many scenarios possible for using 
an ECMS. One field is the company aspect 
where companies try to evaluate the 
knowledge of their employees or to offer pre¬ 
selection tests for new employees. Another 
aspect is the university aspect, which offers 
capabilities to pre-select new tutors or 
employees. Additional the ECMS can be used 
for special educational tasks, like supporting 
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students in self-testing their knowledge in 
bioinformatics, medicine, law or economy. 
ECMS can be used for scholars as well. One 
example are school-contests containing math- 
contests or similar. 

5. Conclusions and future work 

ECMS differs from existing e-Learning, test & 
assessment tools. As difference to common e- 
Learning solution ECMS is not thought only for 
educational purposes. It is correlated to a 
sieving process for the gathering of elite 
students out of a huge collective. Future 
Improvements of ECMS will include time limits 
for each exam. Next there will be support for 
automated exclusion of students after failing n 
homework. Additional the support for 
displaying LATEX2£documents including 
LATEX- and LATEX2e-style formulas should 
be included within future work. Enhanced 
statistical functionalities for better automated 
processes is a necessity. Counting wrong 
given answers will be included. A library 
containing multiple choice tests and homework 
with sorted difficulty have to be build. The 
average solving time for all 19 main exams 
described in section (III) was fast for the given 
software-protection, cryptography and Reverse 
Code Engineering (RCE) topics. For other 
topics like algorithms in bioinformatics, exams 
in medicine or math exams there should be a 
careful preselection of the used exams to 
prevent long sieving times. To protect the 
multiple choice tests from brute forcing attacks 
a randomisation of questions should be 
implemented. 
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